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ABSTRACT 

This final report describes activities and 
accomplishments of a 2-year (1987 through 1989) project on the design 
and evaluation of specific marker and student outcome variables in 
school learning environments for effective mainstreaming of students 
with mild disabilities. The report is organized into eight sections; 
these include a project overview that details the goals and 
accomplishments of each of the project’s three phases and 
descriptions of the seven deliverables. The three phases of the 
project were: (1) identification of major factors and variables 
important to student outcomes; (2) development and validation of the 
Consensus Marker-Outcome Variable System (CMOVS) ; and (3) 
dissemination of findings and use of the CMOVS. The seven 
deliverables included in the report are: (1) a master list of 
variables included in the CMOVS and their definitions; (2) a 
meta-review of the research literature on variables important to 
learning; (3) variables important to learning as rated by 
professionals in the field; (4) research design, measurement 
methodologies, and procedures utilized; (5) a second study of 
variables important to learning; (6) a decision-making framework for 
descriptions of innovative education programs ;. and (7) a checklist 
for description of features of programs that aim to accommodate 
mainstreamed special education students effectively in regular 
education settings, (individual components contain extensive 
references.) (DB) 
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INTRODUCTION 



The purpose of this report is to provide a final summary of activities and 
accomplishments of the project, "Designing and Evaluating School Learning 
Environments for Effective Mainstreaming of Special Education Students: 
Synthesis, Validation, and Dissemination of Research Methods," covering the 
period from September 1, 1987 to December 31, 1989. 

The report is organized in eight sections: a project overview detailing the 
goals and accomplishments of each of the project’s three phases, and 
Deliverables 1-A, 3-A, 3-B, 3-C, 3-D, 3-E and 3-F. (Deliverable 2-A, an Interim 
Progress Report providing a summary of major activities and tasks completed 
during the period of September I, 1998 through August 31, 1989, was completed 
and submitted to OSEP in October 1989). These Deliverables were identified in 
the original proposal and modified based on discussions during the site visit by 
OSEP officials in September 1989. 

The document below outlines the tasks and deliverables that were 
included in our transmittal to OSEP of January 25,1990. 



Tasks Earmarked for Completion During 
the No-cost Extension Period 
(September 1, 1989 - December 31, 1989) 

Four major categories of tasks were targeted for completion during the 
requested no-cost extension period. This document provides: (a) a brief 
description of the specific tasks and deliverables expected to be completed 
during the no-cost extension period, (b) a person loading chart on the amount 
of time the specific project staff would devote to completing the various tasks 
and deliverables, and (c) a budget breakdown for suppjrting the completion of 
the project tasks and deliverables during the no-cost extension period. 



Description of Tasks and Deliverables 
Task 1: 

Complete a master list of definitions of the variables included in the consensus 
marker-outcome variable system (CMOVS). 

Deliverable 1-A: A document providing a brief overview of the purpose 

and design of the CMOVS and definitions of all of the variables included 
in the CMOVS. The listing of the variables and their definitions are 
organized under the six major categories and 30 sub-categories of the 
CMOVS. 

Task 2: 

Complete an Interim Progress Report of the project’s work for submission to 
OSEP. 

Deliverable 2-A: An Interim Progress Report was completed and 

submitted to OSEP. This report provides a summary of major activities 
and tasks completed during the period of September 1, 1988 through 
August 31, 1989. 

Task 3: 

Complete project deliverables (see Table 5 on page 40 of the project’s Year 2 
proposal). The originally proposed deliverables were subsequently modified and 
discussed at the September 7, 1989 site visit. A revised list was included in 
the Appendix of our September 10, 1989 summary of the site visit discussion. 
The following is a list of completed deliverables. 

Deliverable 3-A; A report entitled "Variables Important to Learning: A 

Meta-review of Reviews of the Research Literature." This final report 
was written for the research audience. It incorporates comments and 
suggestions from the technical and stakeholder review panels in response 
to a draft paper reporting the findings from the meta-review. 

Deliverable 3-B: A paper entitled "Variables Important to Learning: A 

Consensus from the Field." This paper reports findings from the survey 



study and discusses the implications of the findings for research and 
practice in improving instruction and learning for students in special 
education programs and students considered at risk of being identified as 
requiring special education services. The final version of the paper 
incorporates the feedback and comments of the technical and stakeholder 
panels. 

Deliverable 3-C: A research paper on the findings of an interview study 

of the instrumentation, measurement procedures, and research design used 
by 30 OSEP funded research projects. This deliverable was based on 
findings from the interview study. The paper provides a summary of the 
research design, measurement methodologies, and procedures utilized by 
senior investigators of research projects funded by OSEP to develop a 
knowledge base for improving learning for special education students 
integrated in regular education settings. 

Deliverable 3-D: A paper summarizing the major findings of the project’s 
work entitled "Variables that are Important to Learning: A Knowledge 

Base for Special and Regular Education." The purpose of this paper is to 
provide an overview of the project and to summarize major findings from 
the meta-review and the consensus from the field based on the survey 
study. The paper is for the practitioner and policy maker audience. 

Deliverable 3-E: A paper providing an illustration of using the CMOVS 

as a framework to communicate the program design emphasis of specific 
programs/practices, and/or as a guiding framework for developing 
programs aiming to provide for the diverse learning needs of special 
education students integrated in regular education settings. This 
deliverable was written for the practitioner audience, particularly 
program developers. 

Deliverable 3-F: A check list for describing and documenting salient 

program features based on the CMOVS framework. This deliverable aims 
to provide an illustration of using the CMOVS framework for systematic 
description and analysis of characteristic features of programs serving 



special education students in regular education settings. It is important 
to note that the purpose of this deliverable is to illustrate an application 
of the CMOVS as a conceptual framework for enhancing communication 
and/or for describing program features and analyzing program 
implementation needs. 

Task 4: 

Dissemination of the work of the project. 

Activity 4.1: Conduct cooperative technical assistance workshops. 

A technical assistance workshop on the CMOVS was held at a regional 
meeting for instructional leaders on October 11-12, 1989. 

Activity 4,2; Develop a strategic plan to disseminate the project’s work 
to researcher and practitioner audiences. 

In addition to targeting specific written products of the project for 
publication in journals with wide circulations to researcher or practitioner 
audiences, the plan also includes submitting proposals for presentations at 
national meetings and finding support for hosting invitational conferences 
with different stakeholder groups (e.g., administrators, teachers, policy 
makers). 



PROJECT OVERVIEW 



The primary task of this project was to obtain consensus among research- 
ers, leading practitioners, policymakers, teacher educators, and parents on 
specific marker and student outcome variables relevant to providing effective 
education for mildly handicapped and learning disabled students in general 
education classroom settings. Mildly handicapped students in the context of the 
project’s work include students classified as educable mentally retarded (EMR), 
learning disabled (LD), and emotionally disturbed (ED), but not psychotic or 
autistic; plus those with limited physical and sensory impairments, but who do 
not have additional severely disabling conditions. The term also encompasses 
students with language development problems (in first or primary language) of 
less than severe degree; many such students are classified in the schools as 
having language or speech problems. 

A basic premise underlying the project design was that important changes 
in the research base are emerging and the extant data base needed to be 
examined to address such instructional design questions as: what aspects of 
school and instruction enhance student learning; what kinds of social 
relationships are important in a learning context for students with mild 
handicaps and learning disabilities in general education classroom environments; 
what characteristics of learners are important in the context of learning for 
students with mild handicaps and learning disabilities in general education 
classroom environments; and, what characteristics of learners are important to 
observe and consider when arranging instruction. 

The overall expected outcome of the project was the development of a 
systematic framework to facilitate increased communication among professionals 
in research, innovative program development, policy making, and the provision 
of education and related services. Increased communication is expected, in 
turn, to lead to improved services for all students, including, and especially, 
those students who require greater-than-usual special education and related 
service support. 



The project’s work was designed to be carried out in three phases. The 



primary task of Phase I was the identification of major factors and variables 
that are important to student learning. The expected outcome of the Phase I 
work was the development of a comprehensive list of marker and outcome 
variables that can be used as guidelines for the design and evaluation of 
programs aimed at providing effective educational and related services for 
students with mild handicaps and learning disabilities in general education 
settings. This work consisted of a two-step process of "synthesizing the 
evidence." The first step involved conducting a comprehensive analysis of the 
extant effectiveness research on teaching and learning and the research on 
special education. The second step of Phase I involved synthesizing evaluations 
of the state of practice by eminent researchers, scholar-practitioners, and 
administrators/instructional leaders. 

The second phase of the project’s work involved the development and 
validation of the Consensus Marker-Outcome Variable System (CMOVS). Among 
the expected outcomes of this aspect of the project’s work were the validation 
of the CMOVS, based on the research literature and expert opinions, and ways 
to use the CMOVS as a framework for improved communication among 
professionals concerned with variables important to learning efficiency and 
effectiveness of students in general, and special education students in general 
education settings. 

The third phase of the project’s work involved the dissemination of the 
findings and use of the Consensus Marker-Outcome Variable System as a means 
of improving communication among researchers, practitioners, policymakers, and 
parents. 



PROJECT ACCOMPLISHMENTS 

The major goals and accomplishments of each phase of the project are 
summarized in the following sections. 



Phase I: Identification and Selection of Specific Variables 
Reported in the Research Literature 



Goals 

The major goal of Phase I work was the development of a marker and 
outcome variable framework that could be used as a guideline for design and 
evaluation of programs aimed at providing effective educational and related 
services for students with mild handicaps and learning disabilities in general 
education settings. The primary task during Phase I of the project was the 
identification of major factors and variables that are important to student 
learning. 

Work by the project staff during the initial six months consisted of a 
two-step process of synthesizing the evidence, namely, (1) the identification of 
major factors and variables that are important to student learning, and (2) the 

synthesis evaluation of the state of practice by eminent researchers, scholar- 

practitioners, and administrators/ instructional leaders. This Phase was 
described in detail in an earlier Interim Report for that period (dated 

September 1, 1987 - February 29, 1988). 

Specifically, the project staff focused on completing the following tasks: 
(a) establishing the Technical Review Panel and the Stakeholder Advisory 
Panels; (b) making the preliminary selection of the synthesis literature; (c) 

developing the preliminary analytic framework; (d) revising the analytic 
framework, based on the feedback from the Technical Review Panel; (e) 
developing the pilot edition of the variable rating scale; and (f) analyzing the 
results of pilot ratings by the four Stakeholder Advisory Panels. 

Accomplishments 

Establish the Technical Review Panel . The Technical Review Panel was 
established at the inception of the project. The primary function of the Panel 
was providing technical advice to the principal investigator and the project 
staff on the development and validation of findings from the research 



synthesis, on the development of the analytic framework, and on the Consensus 
Marker-Outcome Variable Rating System. Criteria for the selection of persons 
to serve on the Technical Review Panel included: (a) national reputations as 
researchers and/or scholar-practitioners in their respective fields, and (b) 
expertise in one or more areas of research on effective teaching and schooling, 
special education, the study of students with special learning needs (i.e., 
students identified as mildly handicapped, learning disabled, academically-at- 
risk, or low-achieving), program evaluation, measurement and instrumentation 
development, and research review and synthesis methodology. 

Select the preliminary synthesis literature . The data source for the 
research synthesis was identified through several means. Office of Special 
Education Programs (OSEP) staff were consulted to develop a list of current 
and recently completed projects funded by OSEP that included major reviews 
of research or research syntheses on themes related to project concerns. 
During the first month of the project, the principal investigator contacted the 
directors of these projects, who provided references to potential data sources. 

ERIC searches also were made for regular and special education research 
and meta-analyses. In addition, members of the Technical Review Panel were 
asked to provide nominations for literature reviews after their first review of 
the preliminary selection of synthesis literature. 



Review the Preliminary selection of synthesis literature. The Technical 
Review Panel reviewed the candidate data sources for the proposed synthesis, 
concentrating on the comprehensiveness and quality of the preliminary list of 
variables that was identified by the project staff. The Panel also commented 
on areas that they judged to be inadequately represented and provided 
additional references for the preliminary list. 

Develop the Preliminary analytic framework . Drawing upon previous 
research reviews and research syntheses such as What Works (U.S. Department 
of Education, 1986), the Handbook of Special Education: Research and Practice 



(Wang, Reynolds, and Walberg, 1987), and the Handbook of Research on 
Teaching (Wittrock, 1986), a preliminary analytic framework of the variables 
important to learning was developed to organize the synthesis data. This 
document incorporates the best thinking from both the general education and 
special education perspectives. 

The categories chosen for the framework were guided by general analytic 
considerations as well as their appropriateness for special education. With 
respect to general considerations, for example, the first-order and second-order 
categories are comprehensive and inclusive, yet as mutually exclusive and 
unduplicative as possible. The category labels are concise and reveal to 
practitioners specific educational actions, while at the same time are 
operationally defined in some detail. The language chosen speaks tellingly to 
general and special educators and avoids jargon whenever possible, while still 
accurately and validly reflecting research content. (It is important to note 
here, however, that readers will be able to pursue any category point included 
in the final framework document -- first to the reviews and syntheses, and 
from these -- if they wish -- to original primary studies.) 

Marker and outcome variables referenced in each of the initial data 
sources were noted. These variables were then grouped with similar variables, 
and after several iterations, seven broad categories were formed. They are: 
(a) state and district context; (b) extra-school context; (c) school level 
characteristics; (d) student characteristics; (e) program design, (f) implementa- 
tion, classroom processes, and classroom climate; and (g) additional variables 
related to learning. 

The initial identification of variables required revision. Some categories 
needed to be expanded, and the organization of the clusters and subclusters of 
variables needed to be refined. However, an effort was made to provide 
mutually exclusive and exhaustive classifications, especially at the levels of the 
seven categories and the major clusters. In constructing the seven categories, 
the major concern was to provide adequate coverage of alterable variables. 
However, a limited number of demographic, or "background," variables were 
included because of their widespread use throughout the literature. 



Review the preliminary analytic framework . The preliminary analytic 
framework was sent to both the Technical Review Panel and to the 
Chairpersons of the four Stakeholder Advisory Panels for feedback. 

Revise the analytic framework. The Delphi Technique was used to develop 
the data base for the design and refinement of the analytic framework. Data 
from the initial feedback (ratings) from the panel of experts (Technical Review 
Panel and chairpersons of the four Stakeholder Advisory Panels) were coded 
and analyzed to serve as the basis for a second round of validation by the 

I 

Panel. 

The results of the individual experts’ ratings and the group mean, as well 
as specific feedback and suggestions from the individual raters, were included 
in the information packet that was sent to the twelve experts who participated 
in the first round of ratings. They were asked to review all of the results 
and comments and decide if they wanted to make changes in their ratings or 
comment on the other experts’ suggestions. Findings from the two rounds of 
ratings were part of the data base for completing the refinement of the 
analytic framework. 

Phase II: Development and Validation of the Consensus 
Marker-Outcome Variable Rating Scale 



Goals 

The focus of the Phase II work was the development of the Consensus 
Marker-Outcome Variable System (CMOVS). The CMOVS was designed as a 
framework for collecting data to describe, monitor, and evaluate program 
implementation. The major goal of the project’s Phase II was a finalized 
version of the Consensus Marker-Outcome Variable System, based on the 
groundwork completed on the preliminary analytic framework during Phase I. 
This work included surveying expert researchers to examine the variables they 
selected as important to learning in their studies and arraying them against 
variables included in CMOVS framework. 
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Accomplishments 

Review the research base . The project staff conducted a systematic 
literature review of the research base on variables that are important to 
learning, using the CMOVS as a guiding framework. Systematic analysis of 179 
hanndbook and annual review chapters, commissioned papers, and other 
authoritative reviews of both regular and special education research literature 
was carried out. A computerized coding scheme was developed to record and 
retrieve information from this review. See Deliverable 3-A, "Variables 
Important to Learning: A Meta-Review of Reviews of the Research Literature." 

Survey the measures and instruments used bv thirty OSEP-funded 
projects . In our original project proposal, one of the deliverables for this 
project was described as including development of a technical report on the 
construct, psychometric and use pattern related to instruments and measures 
used by SEP-funded projects. This proposed task to develop a manual of 
candidate measures and instruments was altered based on concerns and 
recommendations from members of our Technical Review Panel and response 
from senior researchers of the thirty SEP-funded projects (grantees of the 
84.023K. and 84.023FL awards). In addition, there were concerns expressed by 
the OSEP staff on the design and use of such a document. It was felt that 
producing such a document as a reference resource for the field for making 
selection decisions on measures and instrumentation would be subject to 
misinterpretation and strong criticism by researchers and practitioners. These 
changes resulted in delaying completion of this task. 

The principal investigators of the thirty OSEP-funded projects were 
invited to attend a series of conferences to discuss assessment and 
instrumentation issues in the study of the design and effects of school 
programs designed to integrate students with special needs in general education 
settings. The first conference was held at the Temple University Center for 
Research in Human Development and Education in January, 1988. Three 
subsequent conferences were held in conjunction with CEC and AERA annual 
meetings. 

O 
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Based on the discussion and recommendations of participants at these 
conferences, a survey was designed to collect information on measures and 
instruments being used by grantees of the 84.023K and 84.023F1 projects. 

Using a structured interview protocol, data was collected by telephone 
from 31 projects funded for their first year between 1985 and 1988 by the 
Office of Special Education and Rehabilitative Services. Each interview focused 
on questions related to the major variables and measurement methodologies 
utilized by the funded research studies. See Deliverable 3-C, "Research Design, 
Measurement Methodologies and Procedures Utilized by OSEP-Funded Projects" 
for a complete description of the work of developing and validating the 
CMOVS through surveying principal investigators of currently-funded projects. 

Survey the opinions of stakeholders using CMOVS . The survey of opinions 
from stakeholders was sent to six groups of stakeholders: special education 
teachers, general education teachers, principals, school psychologists, 

researchers, and state directors of special education or Chapter 1 programs. 
The first round of surveys was sent out in November 1998; the second round 
was sent out in January 1989; and the third round was sent out in April 1989. 
Findings from the survey are summarized in several deliverables. See 
Deliverable 3-B, "Variables Important to Learning: A Consensus from the 
Field", Deliverable 3-D, "Variables Important to Learning: A Knowledge Base 
for Special and Regular Education", and Deliverable 3-E, "A Decision-Making 
Framework for Description of Innovative Education Programs." 

Review of the drafts of the project’s deliverables . Drafts of the project’s 
deliverables were sent to the project’s Technical Review Panel and stakeholder 
advisory board for review and suggestions. The appendix to this document 
includes the list of reviewers and their comments on the various papers. 



Phase III: Development and Implementation 
of a Broader Dissemination Plan 



Goals 

The primary goal of this final phase of the project was to establish a 
systematic plan for disseminating information about the outcomes of our 
research. 

Our dissemination plan was targeted for three specific audiences: 
researchers and program developers interested in the design, implementation, 
and evaluation of innovative alternatives for accommodating students with 
special needs; practitioners interested in effectively serving mildly handicapped 
and learning disabled students in general education classrooms; and program 
evaluation staff from LEAs and SEAs interested in maximizing their resources 
for providing improved educational services for all students. 

The design of the dissemination plan was based on the premise that 
communication about the research base and practical knowledge of effective 
school programs can only be increased if the project’s findings become known, 
understood, and used. In a real sense, dissemination was a built-in component 
in the work of all three project phases. For example, through the involvement 
of the Technical Review Panel and the Stakeholder Advisory Panel, the 
project’s work has been guided, evaluated, and disseminated by nationally and 
internationally known researchers; federal, SEA, and LEA policymakers; 
practitioners; and teacher educators. These collaborators also constitute an 
ongoing informal network for disseminating information about the project’s 
progress and findings. 

In addition to informal efforts to disseminate information about the 
project, a variety of strategies was used in carrying out our dissemination 
tasks. They included technical assistance workshops, national invitational 
conferences, and presentations of project findings at meetings of professional 
organizations. 



Accomplishments 



Cooperative technical assistance workshops . A series of cooperative 
technical assistance workshops were targeted for research, scholar and 
practitioner audiences. The twofold purpose of these workshops was to discuss 
the major research findings from the project’s work and to obtain feedback on 
the application of the CMOVS. 

A national invitational conference . In addition to the conferences on 
issues and concerns of methodology and instruments for research on 
implementation and efforts of programs designed to serve special needs 
students in the regular education setting, a national invitational conference 
was held in Washington, D.C. in conjunction with the annual meeting on 
educational leadership sponsored by the Institute for Educational Leadership 
(IEL) on December 6 - 9, 1989, to report the work of the project and solicit 
comments and suggestions about alternative ways to disseminate information on 
the project’s findings. We invited professionals in strategic positions and 
asked their advice and assistance in disseminating the work of the project. 
Among the categories of people invited were leading researchers in special 
education and related fields, journal editors, teacher educators and those who 
hold a variety of leadership positions in school systems around the country, 
instructional leaders, and policy makers (superintendents, directors of special 
education, directors of curriculum and instruction, directors of staff 
development, etc.) Holding the conference in Washington, D.C. had the 
additional advantage of allowing us to invite relevant staff members of the 
various professional organizations and whose headquarters are located in 
Washington. It is important to note that although this conference was designed 
as a dissemination activity to solicit feedback on the project’s work from the 
field, project funds were not used to support this conference. 

Develop a systematic Plan to disseminate the project’s written products to 
researcher and practitioner audiences. A systematic plan is being developed for 
submitting the written products of this project to professional journals for 
publications in order to rach a variety of audiences. Among the candidates are 
journals such as American Educational Research Journal. Educational 
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Leadership. Educational Researcher. Elementary School Journal. Exceptional 
Children. Review of Educational Research, and Teaching Exceptional Children . 
Decisions regarding specific journals and topics for submission were based on 
the priority of reaching a variety of stakeholders and maximizing the potential 
of this dissemination strategy. 

The principal investigator and the senior research staff are collaborating 
as co-authors on a summary volume that is based on the final findings of this 
project. The volume will include discussions of all three phases of the work. 
Project staff are exploring the possibilities for publication of the summary 
volume as a book or monograph through one of the national professional 
organizations. 

Present project findings at meetings of professional organizations. Senior 
project staff made presentations on the analysis of our synthesis work at 
regular national meetings of professional organizations. Project staff continue 
to seek opportunities to disseminate the work of the project through 
participation in national and regional meetings. We have submitted proposals 
for presenting the project’s work at relevant professional meetings, even 
though the actual presentations will take place after the project’s ending date. 
The principal investigator has identified other funding sources to support such 
discussants’ activities. 

Id-Cntify specific wa vs the CMQVS can be used as a^Jtuiding framework 
for identifying and describing research variables and for making program 
development and program evaluation decisions . The outcome of this aspect of 
the project’s work was to delineate possible applications of findings from the 
synthesis of the research literature and consensus from the field to facilitate 
increased communication among professionals in research, innovative program 
development, and evaluation of education and related services. 

We identified several ways of applying the CMOVS, based on discussions 
with researchers and practitioners during several presentations of the project’s 
work at national and regional meetings of professional organizations. One 
example is the use of the CMOVS as a design framework for program 
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development and developing aspects of design and implementation of innovative 
programs to provide systematic information base for program selection 
purposes. The CMOVS is currently being used in the design of several studies 
by selected collaborating researchers as the result of cooperative technical 
assistance workshops held by the project staff. Feedback from their work was 
incorporated into a paper on the rationale and design of selected applications 
of the CMOVS as an organizing framework for program development and 
evaluation — see Deliverable 3-E, "A Decision-Making Framework for 
Description of Innovative Education Programs." 

Another example of possible applications of the CMOVS is as a basis for 
a checklist for descriptions of program implementation. Based on the CMOVS 
and a previously designed classroom observation program implementation 
schedule (Reynolds, 1989), a checklist was developed to serve as a guiding 

framework for identifying the presence and absence of critical features of 

effective mainstream learning environments for students with disabilities. This 
instrument was pilot tested by several collaborating researchers in conjunction 
with their studies of classroom environments for effective integration of 
special education services and students in regular education settings. See 

Deliverable 3-F, "A Checklist for Description of Features of Programs that Aim 
to Effectively Accommodate Mainstreamed Special Education Students in 

Regular Education Settings" for the full description and results of this work. 



DELIVERABLES 



1-A "Master List of Variables Included in the CMOVS and Their Definitions" 

3-A "Variables Important to Learning: A Meta-Review of Reviews 
of the Research Literature" 

3-B "Variables Important to Learning: A Consensus from the Field" 

3-C "Research Design, Measurement Methodologies, and Procedures 
Utilized by OSEP-Funded Projects" 

3-D "Variables Important to Learning: A Knowledge Base for Special and 
Regular Education" 

3-E "A Decision-Making Framework for Description of Innovative Education 
Programs" 

3-F "A Checklist for Description of Features of Programs that Aim to 
Effectively Accommodate Mainstreamed Special Education Students in Regular 
Education Settings" 



DELIVERABLE 1-A 



Master List of Variables Included in the CMOVS 
and Their Definitions 



I 
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MASTER LIST OF VARIABLES INCLUDED IN THE CMOVS 
AND THEIR DEFINITIONS 



developed by 



Temple University Center for Research 
in Human Development and Education 



The research reported herein was supported in part by the Temple University 
Center for Research in Human Development and Education and in part by a 
grant from the Office of Special Education Programs of the U.S. Department of 
Education. The opinions expressed do not necessarily reflect the position of 
the OSEP and no official endorsement should be inferred. 



INTRODUCTION 



The consensus marker-outcome variable system, referred to as CMOVS, 
was developed as a result of a comprehensive meta-review and synthesis of 
research on variables considered to be important to learning, and subsequent 
validation based on a survey designed to obtain a consensus from the field. 

The following is a master list of the definitions of the major categories 
of variables that are considered important to learning and the specific 
variables included under each category. The six major categories of variables 
are as follows: (a) State and District Variables, (b) Out of School Contextual 
Variables, (c) School Level Variables, (d) Student Variables, (e) Program Design 
Variables, and (f) Implementation, Classroom Instruction and Climate Variables. 



Master List of Variables and Their Definitions 



Category I. State and District Variables: 

These are variables associated with state and district level school governance and 
administration. They include state curriculum and textbook policies, testing and 
graduation requirements, and teacher licensure; as well as specific provisions in 
teacher contracts, and some district-level administrative and fiscal variables. 



I-A. District Level Demographics and Marker Variables 

1 . School district size 

2. Degree of school district bureaucratization 

3. Degree of school district centralization 

4. Presence of contractual limits on after-school meetings 

5. Limits on class size 

6. Presence of contractual restrictions on activities performed by aides 

7. Degree of central office assistance and support for programs 

8. Degree of board of education support for instructional programs 

9. Per pupil expenditure 

10. Efficiency of transportation system 

I-B. State Level Policy Variables 

1 . Teacher licensure requirements 

2. Degree of state control over textbooks 

3. Degree of state control over curriculum 

4. Academic course and unit requirements 

5. Minimum competency test requirements 

6. Adherence to least restrictive environment/mainstreaming 
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Master List of Variables and Their Definitions 



Category IL Out of School Contextual Variables: 

These are variables associated with the home and community contexts within 
which schools function. They include community demographics, peer culture, 
parental support and involvement, and amount of time students spend out-of- 
school on such activities as television viewing, leisure reading, and homework. 



O-A. Community Variables 

1. Socioeconomic level of community 

2 . Ethnic mix of community 

3. Quality of social services for students 



n-B. Peer Group Variables 

1. Level of peers' academic aspirations 

2 . Level of peers' occupational aspirations 

3. Presence of well defined clique structure 

4. Degree of peers’ substance abuse 

5. Degree of peers' criminal activity 

n-C. Home Environment and Parental Support Variables 

1 . Educational environment (e.g., number of books and magazines at home) 

2. Parental involvement in assuring completion of homework 

3. Parental involvement in assuring regular school attendance 

4. Parental monitoring of student television viewing 

5. Parental participation in school conferences and related activities 

6. Parental application of appropriate, consistent discipline 

7. Parental expression of attention to children 

8. Parental interest in student's school work 

9. Parental expectation for academic success 



D-D. Student Use of Out of School Time Variables 

1 . Student participation in clubs and extracurricular school activities 

2. Amount of time spent on homework 

3. Amount of time spent on leisure reading 

4. Amount of time spent viewing educational television 

5. Amount of time spent viewing noneducational television 
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Master List of Variables and Their Definitions 



Category ID. School Level Variables: 

These are variables associated with school-level demographics, culture, climate, 
policies, and practices. They include demographics of the student body, whether 
the school is public or private, levels of funding for specific categorical pro- 
grams, school-level decision making variables, and specific school-level policies 
and practices, including policies on parental involvement in the school. 



m-A. Demographic and Marker Variables 



1 . 

2 . 

3. 

4. 

5. 

6 . 

7. 

8 . 



Public versus private school 
Size of school 

Level of Chapter I (compensatory education) funding 

Level of Title VII (bilingual) funding 

Level of PL 94-142 (handicapped) funding 

Mix of socioeconomic levels in the school 

Mix of cultural/ethnic groups in the school 

Mix of student language backgrounds in the school 



m-B. Teacher/ Administrator Decision Making Variables 

1. Teacher and administrator consensus on school values, norms, and 
roles 

2. Principal actively concerned with instructional program 

3. Teacher involvement in curricular decision making 

4. Teacher involvement in instructional decision making 

5. Teacher involvement in resource allocation decisions 

6. Teacher involvement in finding ways to increase academic 
performance 






IH-C. School Culture Variables (Ethos Conducive to Teaching and 
Learning) 

1. Use of cooperative, not exclusively competitive, goal structures 

2. School-wide emphasis on and recognition of academic achievement 

3. Low staff absenteeism 

4. Low staff turnover 

5. Low staff alienation 

6. Active collaboration between regular classroom teachers and special 

education teachers 

7. Safe, orderly school climate 

8. Degree of school personnel professional collaboration 
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Master List of Variables and Their Definitions 



Category HI. School Level Variables: (continued) 



m-D. School- Wide Policy and Organizational Variables 

1 . Presence of "effective schools program" 

2. Explicit school grading and academic progress policies 

3. Explicit school-wide discipline policy 

4. Explicit school-wide attendance policy 

5. Coordination of pullout programs for handicapped students with 

regular instructional programs 

6. Use of multi-age grouping 

7. Use of instructional teaming 

8. Use of cross-age tutoring 

9. Use of peer tutoring 

10. Use of academic tracking for specific school subject areas 

1 1 . Minimization of external classroom disruptions (e.g., broadcast 

announcements) 

12. Adherence to least restrictive environment/mainstreaming 

13. Minimum use of suspension and expulsion as discipline tools 

m-E. Accessibility Variables 

1 . Accessibility of educational program (overcoming architectural, 
communication, and environmental barriers 

m-F. Parental Involvement Policy Variables 

1 . Parental involvement in improvement and operation of instructional 

programs 

2. School-sponsored parenting skills workshops (e.g., behavior 

modification, parent effectiveness training) 



Master List of Variables and Their Definitions 



Category IV. Student Variables: 

These are variables associated with individual students themselves, including 
demographics, academic history, and a variety of social, behavioral, motivational, 
cognitive, and affective characteristics. 

IV-A. Demographic and Marker Variables 

1 . Chronological age 

2. Socioeconomic status 

3. Gender 

4. Ethnicity 

5. First or native language 

6. Physical and health status 

7. Special education classifications (e.g., EMR, LD) 

IV-B. History of Educational Placements 

1 . Prior grade retentions 

2. Prior special placements 

3. Current placement in regular class versus self-contained special 

education class 

IV-C. Social and Behavorial Variables 

1. Positive, nondisruptive behavior 

2. Appropriate activity level 

3. Cooperativeness with teacher 

4. Cooperativeness with peers 

5. Ability to make friends with peers 

IV-D. Motivational and Affective Variables 

1 . Attitude toward school 

2. Attitude toward teachers 

3. Attitude toward subject matter instructed 

4. Motivation for continual learning 

5. Independence as a learner 

6. Perseverance on learning tasks 

7. Self-confidence 

8. Academic self-competence concept in subject area instructed 

9. Attributions for success and failure in subject area instructed 
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Master List of Variables and Their Definitions 



Category IV. Student Variables: (continued) 

IV-E. Cognitive Variables 

1. Piagetian stage of cognitive development 

2. Level of reasoning (fluid ability) 

3. Level of spatial ability 

4. Memory 

5. Level of general academic (crystallized) knowledge 

6. Level of specific academic knowledge in subject area instructed 

7. Level of reading comprehension ability 

8. Level of writing ability 

9. Level of computational ability 

10. Level of oral fluency 

1 1 . Level of listening skills 

12. Learning styles (e.g., field independent, visual/auditory learners, 

high cognitive complexity) 

IV-F. Metacognitive Variables 

1 . Self-regulatory, self-control strategies (e.g., control of attention) 

2. Comprehension monitoring (planning: monitoring effectiveness of 

attempted actions; monitoring outcomes of actions; testing, 
revising, and evaluating learning strategies) 

3. Positive strategies for coping with failure 

4. Positive strategies to facilitate generalization of concepts 

IV-G. Psychomotor Variables 

1. Psychomotor skills specific to area instructed 
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Master List of Variables and Their Definitions 



Category V. Program Design Variables: 

These are variables associated with instruction as designed, and with the physical 
arrangements for its delivery. They include the instructional strategies specified 
by the curriculum, and characteristics of instructional materials. 

V-A. Demographic and Marker Variables 

1 . Size of instructional group (whole class, small group, one-on-one 

instruction) 

2. Proportion of students with special needs served in regular classes 

3 . Number of classroom aides required 

4. Resources needed 

V-B. Curriculum and Instructional Variables 

1. Clearly presented academic, social, and altitudinal program goals/ 

outcomes 

2. Use of explicit goal/objective setting for instruction of individual 

student (e.g.. Individualized Educational Plans (IEPs) 

3. Use of mastery learning techniques, including use of instructional 

cues, engagement, and corrective feedback 

4. Use of cooperative learning strategies 

5. Use of personalized instructional program 

6. Use of prescriptive instruction combined with aspects of informal or 

open education 

7. Use of diagnostic -prescriptive methods 

8. Use of computer-assisted instruction 

9. Use of crisis management techniques to control classroom 

disruptiveness 

10. Use of program strategies for favorable affective climate 

1 1 . Alignment among goals, contents, instruction, assignments and 

evaluation 

12. Curriculum units integrated around key discipline-based concepts 

13. Use of multidisciplinary approaches to instructional planning 

(including diagnosis in educational planning) 

14. Presence of information in the curriculum on individual differences 

and commonalities (including handicapping conditions) 

15. Presence of culturally diverse materials in the curriculum 
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Master List of Variables and Their Definitions 



Category V, Program Design Variables, continued 

V-C. Curriculum Design Variables 

1. Materials employ alternative modes of representation 

2. Material is presented in a cognitively efficient manner 

3. Materials employ explicit and specific objectives 

4. Materials employ advance organizers 

5. Materials employ learning hierarchies 

6. Materials are tied to assessment and diagnostic tests 

7. Availability of materials and activities prepared specifically for use 

with whole classroom, small groups, or one-on-one instruction 

8. Degree of structure in curriculum accommodates needs of different 

learners 

9. Student interests guide selection of a significant portion of content 

10. Availability of materials and activities for students with different 

abilities 

1 1 . Availability of materials and activities for students with different 

learning styles 

12. Developmental issues considered 

13. Student experiences considered 
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Master List of Variables and Their Definitions 



Category VI. Implementation, Classroom Instruction, and Climate Vari- 
ables: 

These are variables associated with the implementation of the curriculum and the 
instructional program. They include classroom routines and practices, character- 
istics of instruction as delivered, classroom management, monitoring of student 
progress, and quality and quantity of instruction provided, as well as student- 
teacher interactions and classroom climate. 

VI-A Classroom Implementation Support Variables 

; 1. Creation and maintenance of necessary instructional materials 

2. Adequacy in the configuration of classroom space 

3. Availability of classroom aides 

4. Use of written records to monitor student progress 

5. Establishing efficient classroom routines and communicating rules 

and procedures 

6. Developing student self-responsibility for independent study and 

planning of one's own learning activities 
VI-B Classroom Instructional Variables 

1. Prescribing individualized instruction based on perceived match of 
type of learning tasks to student characteristics (e.g., ability, learning 
style) 

2. Use of procedures requiring rehearsal and elaboration of new 
concepts 

3. Use of clear and organized direct instruction 

4. Systematic sequencing of instructional events and activities 

5. Explicit reliance on individualized educational plans (IEPs) in 

planning day-to-day instruction for individual students 

6. Use of instruction to surface and confront student misconceptions 

7. Use of advance organizers, overviews, and reviews of obejcdves to 

structure information 

8. Clear signaling of transitions as the lesson progresses 

9. Significant redundancy in presentation of content 

10. Teacher conveys enthusiasm about the content 

1 1 . Directing students' attention to the content 

12. Using reinforcement contingencies 

13. Setting and maintaining clear expectations of content mastery 

14. Providing frequent feedback to students about their performance 

15. Explicitly promoting effective metacognitive learning strategies 

16. Promoting learning through student collaboration (e.g., peer tutoring, 

group work) 

17. Corrective feedback in event of student error 

18. Flexible grouping that enables students to work to improve and 

change status/groups 

19. Teaching for meaningful understanding 

20. Degree to which student inquiry is fostered 

21. Scaffolding and gradual transfer of responsibility from teacher to 

student 

22. Degree to which assessment is linked with instruction 

23. Skills taught within the context of meaningful application 

24. Good examples and analogies to concretize the abstract and 

familiarize the storage 

25. Consideration of the teacher’s use of language in the instructional 

process 

26. Explicitly promoting student self-monitoring of comprehension 
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Master List of Variables and Their Definitions 







Category VI. Implementation, Classroom Instruction, and Climate Variables: (continued) 



VI-C. Quantity of Instruction Variables 



1. Length of school year 

2. Length of school day 

3. Time on task (amount of time students are actively engaged in 

learning) 

4. Time spent in direct instruction on basic skills in reading 

5. Time spent in direct instruction on basic skills in mathematics 

6. Time allocated to basic skills instruction by regular classroom 

teacher 

7. Time allocated to basic skills instruction by special education teacher 

8. Difference between academic learning time and allocated learning 

time 

9. Time spent out of school on homework 

10. Time spent out of school viewing educational television 

1 1 . Time spent out of school in informal learning experiences (e.g., 

museum trips, scouts) 

1 2. Nature of regular classroom content missed by students during 

participation in pullout programs 



Vl-D. Classroom Assessment Variables 



1 . Use of assessments to create detailed learner profiles rather than 

simple classifications or unlaborated total scores 

2. Use of assessment as a frequent, integral component of instruction 

3. Accurate, frequent measurement of basic skills in reading 

4. Accurate, frequent measurement of basic skills in mathematics 



VI-E. Classroom Management Variables 

1. Minimal disruptiveness in classroom (e.g., no excessive noise, no 

students out of place during instructional activities, no destructive 
activities) 

2. Group alerting (teaching uses questioning/recitation strategies that 

maintain active participation by all students) 

3. Learner accountability (teacher maintains student awareness of 

learning goals and expectations) 

4. Transitions (teacher avoids disruptions of learning activities, brings 

activities to a clear and natural close, and smoothly initiates new 
activity) 

5. Teacher "withitness" (teacher is continually aware of events and 

activities and minimizes disruptiveness by timely and nonconfron- 
tational actions) 
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Master List of Variables and Their Definitions 



Category VI. Implementation, Classroom Instruction, and Climate Variables: (continued) 

VI-F. Student and Teacher Interactions: Social Variables 

1 . Student initiates positive verbal interactions with other students and 

with teacher 

2. Student responds positively to questions from other students and 
from teacher 

3. Teacher reacts appropriately to correct and incorrect answers 

4. Teacher reinforces positive social interactions with students rejected 

by peers 

5. Teacher provides explicit coaching on appropriate social behaviors 

6. Teacher provides explicit coaching to reduce aggression 

VI-G. Student and Teacher Interactions: Academic Variables 

1 . Teacher asks academic questions frequently 

2. Teacher asks questions predominantly low in difficulty 

3 Teacher asks questions that are predominantly low in cognitive level 

4. Teacher maintains high post-question wait time 

5. Frequent calls for extended, substantive oral and written response 

(not one-word answers) 

VT-H. Classroom Climate Variables 

1 . Cohesiveness (members of class are friends sharing common 

interests and values and emphasizing cooperative goals) 

2. Low friction (students and teacher interact in a considerate and 

cooperative way, with minimal abrasiveness) 

3. Low cliqueness (students work with many different classmates, and 

not just with a few close friends) 

4. Satisfaction (students are satisfied with class activities) 

5. Speed (the placing of instruction is appropriate for the majority of the 

students) 

6. Task difficulty (students are continually and appropriately 

challenged) 

7. Low apathy (class members are concerned and interested in what 

goes on in the class) 

8. Low favoritism (all students are treated equally well in die class, and 

given equal opportunities to participate) 

9. Formality (students are asked to follow explicitly stated rules 

concerning classroom conduct and activities) 

10. Goal direction (objectives of learning activities are specific and 

explicit) 

1 1 . Democracy (all students are explicitly involved in making some 

typies of classroom decisions) 

12. Organization (class is well organized and well planned) 

13. Diversity (the class divides its efforts among several different 

purposes) 

14. Environment (needed or desired books and equipment are readily 

available to students in the classroom) 

15. Comp>etition (students comptete to see who can do the best work) 



DELIVERABLE 2-A 

Interim Progress Reports: Delivered to OSEP 



NOTE: These reports are not included here, since they were submitted to 

OSEP in February 1988 and October 1989, respectively. 
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Abstract 



This paper reports a comprehensive "meta-review" and synthesis of 
research on variables related to learning, including both cognitive and affective 
schooling outcomes. A conceptual framework was developed, encompassing 228 
items related to school learning, organized a Priori into 30 scales within six 
categories. Search and selection procedures yielded 179 selected handbook and 
annual review chapters, commissioned papers, and other authoritative reviews. 
Content analysis yielded over 3,700 ratings of the strength of influence of the 
variables on learning. They confirm the primacy of student, classroom, home, 
and community influences on learning relative to more distal policy variables 
such as state and district characteristics. They also highlight the importance 
of metacognition, classroom management, quantity of instruction, classroom 
interactions, classroom climate, and the peer group. 
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Variables Important to Learning: 
A Meta-Review of Reviews of 
the Research Literature 



Educational research has identified a large number of variables related to 
school learning. Indeed, such a multiplicity of distinct influences on 
achievement have been found that educators may be perplexed as to which 
items are most important. Educational researchers, policy makers, and 
practitioners all require clearer guidance concerning the relative importance of 
different learning influences and the particular variables most likely to 
maximize school learning. To address this need, a comprehensive review and 
synthesis of handbooks, review annuals, and other highly synthetic prior 
reviews was undertaken. Its purpose was to characterize the most 
authoritative scholarly opinion about ways to optimize educational outcomes 
across a range of educational conditions and settings. This research synthesis 
is distinguished by its comprehensiveness, its orientation toward practical 
school improvement strategies, and its focus on comparing the relative 
contributions of different items to learning. To organize the synthesis, a 

conceptual framework was developed which draws heavily upon major 
theoretical models of school learning. Before turning to this framework, the 
evolution of these earlier theoretical models is briefly described. 

Evolution of Models of School Learning 

J. B. Carroll (1963) introduced educational researchers to models of school 
learning in his Teachers College Record aptly entitled article, "A Model of 
School Learning." In his model, he put forth six constructs: aptitude, ability 

to comprehend instruction, perseverance, clarity of instruction, matching the 
task to student characteristics, and opportunity to learn. These constructs, 
which succinctly capture the psychological influences on school learning, 
became a point of departure for other models to follow. The 1960s and 1970s 
were marked by the introduction of several additional important models of 
learning, including those of Bruner (1966), Bloom (1976), Harnischfeger and 
Wiley (1976), Glaser (1976), and Bennett (1978). 
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All of these models recognize the primary importance of student ability, 
and included constructs such as aptitude, prior knowledge, verbal IQ, and pupil 
background. Most of them also address the importance of motivation, by 

employing such constructs as perseverance, self concept of the learner, and 
attitude toward school subject matter. This acknowledgement of individual 
difference variables among learners stood in contrast to more narrowly 
psychological studies of influences on learning, which generally treated 
individual differences as a source of error, focusing instead on 
instructional-treatment variables (Hilgard, 1964). 

In addition to student variables, each of the models of school learning 
noted above also gave salience to constructs developed from studies of 
classroom instruction. These constructs varied in generality, some being as 
broad as "instructional events" or "clarity of instruction," and others as narrow 
as "use of cues" or "feedback and correctives." 

Although later models brought some refinement in the ways in which 
individual difference variables and instructional variables were defined and the 
ways in which they were related to one another, the primary contributions of 
more recent models have been in extending the range of influences considered. 
Haertel, Walberg, and Weinstein (1983), for example, identified nine theoretical 
constructs that exhibit consistent causal influences on academic learning: 
student age or developmental level, ability (including prior achievement), 
motivation, amount or quantity of instruction, quality of instruction, 
psychological environment of the classroom, influence of the home, influence 
of the peer group outside of school, and exposure to mass media. They 
showed that previous models of school learning neglected extramural and 
social-psychological influences. 

The evolution of models of school learning was further advanced with the 
introduction of models of adaptive instruction (Wang & Lindvall, 1984; Wang & 
Walberg, 1985). School-based implementation of models of adaptive instruction 
are designed to help schools create learning environments that maximize each 
student’s opportunities for success in school. It paid particular attention to 



new variables associated with instructional delivery systems, program design, 
and implementation. It attended in particular to those features that Glaser 
(1982) referred to as the "large practical variables," and included efficient 
allocation and use of teacher and student time, a practical classroom 
management system, systematic teacher feedback and reinforcement of student 
learning behavior and progress, instructional interactions based on the 
diagnosed learning needs of individual students, and flexible administrative and 
organizational patterns responsive to program implementation and staffing 
needs. 

Another contribution to models of school learning came from sociologists 
concerned with the identification of effective schools. Ronald Edmonds (1979) 
is most strongly associated with this identification of variables associated with 
exceptionally effective schools, especially for the urban poor. Significant 

contributions to effective schools models were also made by Brookover (1979), 
Brookover and Lezotte (1979), and Rutter, Maughan, Mortimore, Ouston, and 
Smith (1979). Illustrations of the types of variables characterizing effective 
schools include degree of curriculum articulation and organization, schoolwide 
staff development, parental involvement and support, schoolwide recognition of 
academic success, maximized learning time, district support, clear goals and 
high expectations, orderly and disciplined school environment, and leadership of 
principal characterized by attention to quality of instruction (Purkey & Smith, 
1983). 

These various models of school learning all contribute a variety of items, 
or variables, that may be useful to educational practitioners. Individual 

researchers may focus their work on particular variables or constructs, but the 
purpose of this synthesis was to try to provide a synoptic view of the entire 
panoply of variables. 



Methods and Procedures 

The first step in developing the meta-review described in this paper was 
to delineate a comprehensive set of variables organized into an inclusive 
conceptual framework.. Next, a corpus of over 150 books, book chapters. 
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reports, and other sources was identified. The 228 items in the conceptual 
framework were listed on a detailed, fifteen-page coding form, and each of the 
sources was then coded using that form. In all, over 2,500 pages of coding 
forms were completed. Each citation or discussion of an item influencing 
learning outcomes was coded by page number, together with a notation of the 
reported strength of its influence on learning. These detailed text citations by 
page number have been placed in an archive. 2 

The detailed ratings were then recoded onto a set of summary forms, one 
for each chapter or other source, which gave overall ratings of strength of 
influence for each of the items discussed in that source. These summary 
ratings were entered into machine-readable files and analyzed to determine the 
emergent consensus on which items exert the most powerful influence on 
learning outcomes. The initial coding tabulated well over 10,000 separate 
statements in the research literature concerning of the strength of association 
between one of the 228 items and learning outcomes. These were reduced to 
over 3,700 summary ratings, which were then keyed and analyzed. 3 

Before describing the data analyses and the findings of this study, the 
development of the conceptual framework, selection of the corpus of studies, 
and coding procedures are briefly described. 

Conceptual framework for items related to learning 

The identification of a comprehensive set of items began with a close 
examination of the models of school learning described above, as well as 
selected sources, including Brophy (1986), Keogh, Major-Kingsley, 
Omori-Gordon, and Reid (1982), Wang and Walberg (1985), and Wittrock (1986). 
Potential variables were written on separate index cards, then consolidated and 
organized into a preliminary version of the final coding scheme. This draft 
coding scheme was sent to members of the Scientific Advisory Panel of the 
Center for Research in Human Development and Education at Temple 
University 4 . Based on detailed commentaries received from the Panel members, 
the framework was revised to include four additional items, and to improve its 
organization. 



The final framework organized the 228 items related to learning into 30 a. 
priori scales within six broad categories. The six categories were ordered 
roughly from more distal to more proximal factors. Brief descriptions of the 
categories are presented in Table 1, together with illustrative items from each 
scale. 

Selection of a corpus of studies 

A vast research literature addresses one or more of the potential learning 
influences represented by the conceptual framework, and it clearly would not 
be possible to examine all of the thousands of original studies relevant to a 
synthesis of this scope. Indeed, even the literature of review articles is 
massive. For this reason we focused on authoritative reviews and handbook 
chapters, especially those sponsored by the American Educational Research 
Association and other organizations, and selected additional syntheses in 
government documents and other sources. A preliminary list of sources was 
reviewed by the Scientific Advisory Panel, and revised following their 
recommendations. Following this review, the sources chosen included chapters 
from the past decade or more of the Review of Research in Education, the 
Annual Review of Psychology, and the Annual Review of Sociology, as well as 
the Handbook of Research on Teaching (Wittrock, 1986), Designs for 
Compensatory Education (Williams, Richmond, & Mason, 1986), more specialized 
handbooks, and a small number of journal articles chosen to assure coverage of 
all of the areas addressed in the comprehensive framework. Initially, over 200 
articles, chapters, and other sources were identified. All of these sources 
were read, but some were excluded from the final corpus because they failed 
to address K-12 instruction in regular school settings, because they addressed 
exceptionally narrow and atypical learning outcomes, or because they were 
relevant only to rare or special-learner populations. 

A total of 179 sources were included in the final corpus of studies (see 
Appendix for a complete bibliography). All of these were relevant to a range 
of cognitive and or affective learning outcomes for K-12 learners in formal 



educational settings. Table 2 presents a summary by type of the source 
documents included in the final synthesis. 

Coding procedures 

Each source document was coded initially onto a detailed rating form, 
which allowed for the recording of multiple references in a single document to 
the same item. In addition to coding references to the 228 prespecified items, 
space was provided for the coding of any additional items related to learning 
outcomes, referred to on the form as supplementary items. Brief notes were 
also recorded for most sources, including page references, comments on the 
source’s overall relevance, and any limitations on the learner populations 
and/or varieties of learning outcomes addressed. This archived documentation 
has been retained by the first author. 

Each reference to an item’s relation to learning outcomes was coded on a 
three-point scale, with "1" representing a weak, uncertain, or inconsistent 
relation to learning; ’’2” representing a moderate relation; and ”3'' representing 
a strong relation. Where "vote counts" or proportions of confirming studies 
were reported, a "3" indicated that more than 80 percent of the studies 
discussed had found a statistically significant association of an item to 
achievement; a "2" indicated that between 40 percent and 80 percent of the 
studies found support for the relationship; and a "1" indicated less than 40 
percent in support. Where results were summarized in terms of effect sizes, a 
code of "3" was assigned to effect sizes greater than .33, "2" to effect sizes of 
.10 to .33, and "1" for smaller effect sizes. Where correlations were reported, 
"3" was used for correlations greater than .40, "2" for correlations of .15 to 
.40, and "1" otherwise. 

In many cases, the source documents did not present quantitative indices 
like effect sizes or correlations, and so it was necessary to judge the strength 
of the evidence presented from prose descriptions of the conclusions from 
bodies of research. . In these cases, the strength of the evidence presented was 
judged weak, moderate, or strong, and coded accordingly. Even though all of 
the 228 items were defined in such a way that they were expected to relate 



positively to learning, there were rare instances in which negative conclusions 
from the literature were reported. 5 

Following the coding of all specific references by page number, ratings 
were transcribed onto a second, summary form for each source, prior to keying 
for data analysis. At this stage, a single, summary code — the average of all 
the ratings for each source document -- was recorded indicating the strength 
of association for each item discussed in the source, according to the 
preponderance of the specific references noted. 6 

Data Analysis 

After inspecting univariate frequency distributions for each of the 228 
separate items to assure that no values were out of range, the separate items 
were aggregated to the level of the thirty scales described in Table 1. This 
was accomplished by taking the average of all non-missing values in a scale, 
for each source. In cases where a source document did not discuss any of the 
separate items in a scale, a missing data code was entered. In those rare 
cases where negative findings were coded, their negative signs were retained 
when averages were taken. 

In a second stage of data reduction, six additional variables were created, 
corresponding to the categories described in Table 1. The values of these 
variables for each source were weighted averages of all nonmissing scale values 
comprised by that category. 7 Means, standard deviations, and alpha 
reliabilities for the six categories and thirty scales are presented in Table 3. 
The reliabilities for documents (not raters) range from .71 to .99. All but four 
are greater than .80, and most exceed .90®. 

Table 3 also reports the number of sources that discussed items in each 
scale. Surprisingly, the frequency with which different scale items are 
discussed in the literature is not a reliable guide to their importance for 
learning outcomes. The Spearman rank correlation between frequencies and 
means across the 30 scales is only .10. 



educational settings. Table 2 presents a summary by type of the source 
documents included in the final synthesis. 

Coding procedures 

Each source document was coded initially onto a detailed rating form, 
which allowed for the recording of multiple references in a single document to 
the same item. In addition to coding references to the 228 prespecified items, 
space was provided for the coding of any additional items related to learning 
outcomes, referred to on the form as supplementary items. Brief notes were 
also recorded for most sources, including page references, comments on the 
source’s overall relevance, and any limitations on the learner populations 
and/or varieties of learning outcomes addressed. This archived documentation 
has been retained by the first author. 



Each reference to an item’s relation to learning outcomes was coded on a 
three-point scale, with "l" representing a weak, uncertain, or inconsistent 
relation to learning; "2" representing a moderate relation; and "3" representing 
a strong relation. Where "vote counts" or proportions of confirming studies 
were reported, a "3" indicated that more than 80 percent of the studies 
discussed had found a statistically significant association of an item to 
achievement; a "2" indicated that between 40 percent and 80 percent of the 
studies found support for the relationship; and a "1" indicated less than 40 
percent in support. Where results were summarized in terms of effect sizes, a 
code of "3" was assigned to effect sizes greater than .33, "2" to effect sizes of 
.10 to .33, and "1" for smaller effect sizes. Where correlations were reported, 
"3" was used for correlations greater than .40, "2" for correlations of .15 to 
.40, and "1" otherwise. 

In many cases, the source documents did not present quantitative indices 
like effect sizes or correlations, and so it was necessary to judge the strength 
of the evidence presented from prose descriptions of the conclusions from 
bodies of research. In these cases, the strength of the evidence presented was 
judged weak, moderate, or strong, and coded accordingly. Even though all of 
the 228 items were defined in such a way that they were expected to relate 
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for data analysis. At this stage, a single, summary code •• the average of all 
the ratings for each source document -- was recorded indicating the strength 
of association for each item discussed in the source, according to the 
preponderance of the specific references noted. 8 

Data Analysis 

After inspecting univariate frequency distributions for each of the 228 
separate items to assure that no values were out of range, the separate items 

were aggregated to the level of the thirty scales described in Table 1. This 

was accomplished by taking the average of all non-missing values in a scale, 

for each source. In cases where a source document did not discuss any of the 

separate items in a scale, a missing data code was entered. In those rare 
cases where negative findings were coded, their negative signs were retained 
when averages were taken. 

In a second stage of data reduction, six additional variables were created, 
corresponding to the categories described in Table 1. The values of these 
variables for each source were weighted averages of all nonmissing scale values 
comprised by that category. 7 Means, standard deviations, and alpha 
reliabilities for the six categories and thirty scales are presented in Table 3. 
The reliabilities for documents (not raters) range from .71 to .99. All but four 
are greater than .80, and most exceed .90 s . 

Table 3 also reports the number of sources that discussed items in each 
scale. Surprisingly, the frequency with which different scale items are 
discussed in the literature is not a reliable guide to their importance for 
learning outcomes. The Spearman rank correlation between frequencies and 
means across the 30 scales is only .10. 



Summary of Results 



Table 3 shows the importance of many distinct influences on school 
learning outcomes. Over all 30 scales, the mean rating was roughly 1.8, a 
little below the level designated "moderate relation to learning." More 
important, however, the synthesis shows which categories, scales, and specific 
items are most strongly associated with learning outcomes. In discussing 
results by category and by scale, relevant findings concerning specific items 
will be presented to clarify or elaborate the category- and scale-level findings 
reported in Table 3. 

At the highest level of generality, this synthesis confirms the importance 
of the quality of schooling for learning outcomes. Of the six categories, the 
highest ratings overall were assigned to "Program Design Variables," followed 
by "Out of School Contextual Variables." The category reflecting the quality 
of instruction as delivered, "Classroom Instruction and Climate Variables," 
ranked third in importance, closely followed by "Student Variables." The last 
two categories, "School Level Variables" and "State and District Variables," 
received markedly lower ratings overall. This overall ranking of sources of 
influence contrasts sharply with the "conventional wisdom" since the time of 
the Equality of Educational Opportunity (EEO) Survey (Coleman, et al., 1966) 
that quality of schooling has relatively little impact on schooling outcomes 
relative to out-of-school, socioeconomic variables. 

The importance of proximal psychological variables may be seen in the 
scales that obtained the highest ratings. 9 Those scales with mean ratings of 
2.0G or greater were (beginning with the highest): 



0 


Metacognition 


X = 2.08 


0 


Classroom Management 


X - 2.07 


0 


Quantity of Instruction 


X = 2.02 


0 


Student/Teacher Interactions: Social 


X - 2.02 


0 


Classroom Climate 


X « 2.01 


0 


Peer Group Influences 


X = 2.00 



In the remainder of this section, the categories and scales are discussed 
in turn, and those scales and items that received exceptionally high ratings are 
highlighted. The categories representing instruction as designed and 



instruction as delivered are discussed first. These are followed by 
out-of-school context and student characteristics. Finally, the more distal 
variable categories of school level variables and state and district variables are 
addressed. 

Program Design Variables 

This category includes instruction as designed, and the physical 
arrangements for its delivery, organized into three scales, as shown in Tables 
1 and 3. The scale "Demographic and Marker Variables" was rated highest of 
the three, and within this scale the most highly rated items are "size of 
instructional group (whole class, small group, or one-on-one instruction)," 
"number of classroom aides," and "resources needed." (Ratings for these items 
ranged from 1.95 to 2.00.) Thus, the most important aspect of program design 
appears to be the intensity of educational services provided to each learner. 
More aides, smaller groups, or increased material resources are associated with 
significantly higher learning outcomes. 

"Curriculum and Instructional Variables" includes a number of items with 
average ratings above 2.0 (moderate relation to learning). The highest rated 
of these suggest that the key to effective instructional design is the flexible 
and appropriate use of a variety of instructional strategies, while maintaining 
an orderly classroom environment. The highest overall rating in this scale was 
for "Use of ... techniques to control classroom disruptiveness." This item was 
followed by "use of prescriptive instruction combined with aspects of informal 
or open education" and "presence of information in the curriculum on 
individual differences and commonalities," both of which explicitly relate to 
student diversity and individualization. Other highly rated items referred to 
specific instructional strategies, including "use of mastery learning techniques, 
... instructional cues, engagement, and corrective feedback, " "use of 
cooperative learning strategies," and "use of diagnostic-prescriptive methods." 

"Curriculum Design" also includes several items with average ratings near 
2.0, although none exceeds the "moderate" level. High ratings were given to 
"materials employ alternative modes of representation" and "degree of structure 



instruction as delivered are discussed first. These are followed by 
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student diversity and individualization. Other highly rated items referred to 
specific instructional strategies, including "use of mastery learning techniques, 
... instructional cues, engagement, and corrective feedback, " "use of 
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in curriculum accommodates needs of different learners," both of which 
reinforce the importance of offering a variety of instructional materials and 
approaches to accommodate individual differences. The importance of the 
organization of curriculum content is revealed by two highest-rated items in 
this scale, "materials employ learning hierarchies” and "material is presented in 
a cognitively efficient manner." 

Implementation. Classroom Instruction, and Climate Variables 

This category includes support of the curriculum and the instructional 
program; classroom routines; specific instructional, assessment, and classroom 
management practices; quantity of instruction; academic and nonacademic 
student-teacher interaction; and classroom climate. It is by far the largest of 
the six categories, comprising 79 of the 228 items and eight of the thirty 
scales. Half of these scales had mean ratings above 2.00, placing them among 
the most influential scales overall. 

High ratings in the areas of Implementation, classroom instruction, and 
climate again point up the importance of maintaining an orderly classroom 
environment and providing clear, well organized instruction appropriate to the 
needs of individual learners. In the overall ranking of all 30 scales, 
"Classroom Management" ranked second. Its most critical items were "group 
alerting (teacher uses questioning/recitation strategies that maintain active 
participation by all students)” and "learner accountability (teacher maintains 
student awareness of learning goals and expectations)." Smooth transitions 
from one instructional activity to another, minimal disruptions, and teacher 
awareness of what is going on in the classroom at all times also received mean 
ratings above 2.00. 

"Quantity of Instruction" was ranked third overall, following "Classroom 
Management." It includes time spent in direct instruction, especially direct 
instruction on basic skills; time spent on homework; and length of the school 
day and the school year. The importance accorded quantity of instruction is 
not surprising. This construct has appeared in many of the most widely cited 
models of school learning (Haertel, Weinstein, & Walberg, 1983). 
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"Student/Teacher Interactions: Social" ranked fourth overall, and 

"Classroom Climate" was ranked fifth. The high ranking for social interactions 
was almost entirely due to just two items with mean ratings of 2.00 or greater: 
"teacher reacts appropriately to correct and incorrect answers," and "student 
responds positively to questions from other students and from teacher." 10 
"Classroom climate" included fifteen items with ratings of 2.00 or greater. 
Taken together, the highly rated items in these two scales characterize a 
classroom in which teacher and students interact considerately and 

cooperatively, where students work with several classmates, share common 
interests and values, and pursue cooperative goals. Students are actively 
engaged in learning, and are involved in making some types of classroom 
decisions. At the same time, the class is well organized and well planned, 
with a clear academic focus. Objectives of learning activities are specific and 
explicit, and students feel continually and appropriately challenged, with the 
pacing of instruction appropriate for the majority. 

The remaining scales under "Implementation, Classroom Instruction, and 
Climate Variables" have much lower overall ratings, but include more than 
twenty specific items with means of 2.00 or greater. The majority of these 
items refer to instructional organization, and to mechanisms for assuring that 
students understand that organization and the goals of instruction. For 
example, high ratings were given to use of advance organizers and directing 
students’ attention to the content to be learned; and to clear and organized 
direct instruction, systematic sequencing of lesson events, and clear lesson 
transitions. Other highly rated items included corrective feedback in case of 
student error, frequent academic questions, and accurate measurement of skills. 
Finally, the literature strongly supports the teaching of skills in the context of 
meaningful application, use of good examples and analogies, and teaching for 
meaningful understanding, together with explicit promotion of student 
self-monitoring of comprehension and gradual transfer of responsibility for 
learning from the teacher to the student. 
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Extramural Variables 

This category includes items associated with the home and community 
contexts within which schools function. As presented above, "Peer Group 
Variables” was ranked sixth among all scales. This was due primarily to the 
emphasis placed on peers’ educational and occupational aspirations, both of 
which had mean ratings of 2.00 or higher. 

Additional highly rated items in this category reflected parental interest 
and involvement in students’ school work. For example, 'parental involvement 
in assuring completion of homework,” 'parental participation in school 
conferences and related activities,” and 'parental interest in students’ school 
work" all received high ratings. The educational environment of the home 
(e.g., number of books and magazines) was also cited in numerous sources, and 
received consistently high ratings. Student participation in clubs and 
extracurricular school activities and time spent on leisure reading were also 
moderately related to learning outcomes. 

Student Variables 

These are items associated with individual students themselves, including 
demographics, academic history, and various social, cognitive, and affective 
characteristics. Among these items, the one with the highest rating was 
"psychomotor skills specific to area instructed," with a rating of 2.33. This 
was the only item included in the "Psychomotor Variables" scale. However, as 
explained above, this mean is based on only six sources. It is best regarded as 
a statistical artifact, and will not be further discussed. 

"Metacognitive Variables" received the highest mean ratings of any of the 
remaining scales in the entire framework. Highly rated metacognitive items 
include "comprehension monitoring (planning; monitoring effectiveness of 
attempted actions; testing, revising, and evaluating learning strategies)," 
"self-regulatory, self-control strategies (e.g., control of attention)," and 
"positive strategies to facilitate generalization of concepts." 
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A number of specific items in the remaining "Student Variables" scales 
also had high ratings, including positive behavior and ability to make friends 
with peers, motivation for continual learning, and perseverance on learning 
tasks. Highly rated items from the "Cognitive" scale included several 
representing general mental abilities, levels of basic skills sufficient to profit 
from instruction, and prior knowledge in the subject area instructed. 

School Level Variables, and State and District Variables 

Educational policy items at the school, district, and state levels appear 
from this research synthesis to have relatively little association with learning 
outcomes, as shown by low mean ratings for categories and scales. A few 
items in this area received mean ratings of 2.00 or higher, but nearly all of 
these were based on fewer than ten sources. Nonetheless, several school-level 
educational practices emerged as important. These included the presence of an 
"effective schools” program; explicit school grading, academic progress, and 
attendance policies; and a safe and orderly school climate. Peer and cross-age 
tutoring, which were classified as school-level variables when their 
implementation required coordination among self-contained classrooms, also 
received moderate or higher ratings based on discussions in more than ten 
sources. 



Discussion 

This research synthesis confirms that distal policy variables are less 
important to schooling outcomes than quantity and quality of instruction, home 
environment, or student characteristics. Of the six categories in the 

conceptual framework (See Tables 1 and 3), "State and District Variables" and 
"School Level Variables," both comprising mainly policy variables, had markedly 
lower mean ratings than the remaining four categories. The items most 
important to learning outcomes are those that are directly tied to students' 
engagement with the material to be learned. 

In contrast to the earlier view that quality of schooling is of little 
importance relative to out-of-school factors (e.g., Coleman, et al., 1966), this 



synthesis also suggests that from kindergarten through the twelfth grade, 
across a range of content areas and educational contexts, quality and quantity 
of instruction are roughly equal in importance to student characteristics and 
out-of-school contextual items. 

Furthermore, the present synthesis of educational research is considerably 
more comprehensive than What Works: Research About Teaching and Learning, 
the widely-distributed pamphlet of the U.S. Department of Education (1986); it 
contains both highly effective and relatively less effective practices. The 
present synthesis, moreover, draws on a larger body of literature, and contains 
a more explicit methodology that can be replicated by other investigators. It 
contains some 228 practices in comparison with 41 in the original What Works 
(and 62 in the second edition); and it gives a numerical rating to each one as 
well as composites. Yet none of the findings of What Works and the present 
work are discordant. What Works contains specific findings and elaborates on 
and illustrates various techniques. Such techniques are described specifically 
enough to be understood by parents and teachers; perhaps they might even be 
put into practice without assistance. 

To be useful to practitioners, the present findings, many of which are 
abstract and concern more complex practices, would have to be further 
described and exemplified. To accomplish this, reviewers would have to return 
to the review literature and perhaps the original studies to analyze the 
specific operational definitions of techniques. These would require translation 
into plain language and prescriptive practices. Many are sufficiently complex 
that they could not be implemented without training and staff development. 
Such an effort would be considerable but worthwhile. 

Turning from the level of the six broad categories to the thirty scales, 
those identified as most important to good learning outcomes are student 
metacognition, effective classroom management, quantity of instruction, positive 
and productive student/teacher Interactions, a classroom climate conducive to 
learning, and a peer culture supportive of academic achievement. These broad 
conclusions are supported by a number of more specific findings from the 
research synthesis. These selected findings are highlighted below. 



Student characteristics 



Individual differences among students have long been recognized as 
critical determinants of learning outcomes, but it was both surprising and 
encouraging that in this synthesis the metacognitive items emerged as most 
important, including comprehension monitoring, use of self-regulatory, 
self-control strategies, and use of strategies to facilitate generalization of 
concepts. Metacognitive variables are heavily cited in the current literature, 
in contrast to an earlier focus on relatively stable general mental abilities. A 
better understanding of these alterable variables may ultimately help the great 
majority of students to reach higher achievement levels through appropriate 
training in metacognition. Two additional student items accorded importance 
in the research literature were "perseverance on learning tasks" and 
"motivation for continual learning." Both of these reinforce the conclusion 
that consistent engagement with the subject matter to be learned is critical to 
school success. 

Quality and quantity of instruction 

Classroom management and climate and student-teacher interactions 
represent an important constellation of variables related to effective 
instruction. Detailed examination of the highly rated items in these areas 
reveals a portrait of cooperative, cohesive, goal-directed classrooms in which a 
variety of educational approaches and activities are employed. Items heavily 
cited in the research literature include sound organization and systematic 
sequencing of instruction, and effective use of direct, teacher-centered 
instruction. Among other instructional approaches frequently linked to positive 
learning outcomes were peer and cross-age tutoring and cooperative group 
learning strategies. 

Several items associated with quantity of instruction also emerged as 
important, including student time on task, length of school day and school 
year, amount of time allocated to direct instruction in basic skills, and time 
spent out of school on homework and on leisure reading. Of these, the most 



frequently cited variable is time on task. These time-related variables have 
clearly become well established and widely accepted as determinants of learning 
outcomes, in spite of criticisms cited by (Shulman, 1986) of time as an "empty 
vessel." 

Out of school context 

There has been increasing attention in the research literature to the role 
of parental involvement and support variables in promoting student learning. 
The synthesis affirmed the importance of these items, as well as peer group 
influences. These findings were reflected in ratings for parental involvement 
in school activities, interest in schoolwork, and monitoring of school 
attendance and homework completion. Parental support might also be mediated 
through influence on students’ selection of friends. Peer group variables, 
especially academic and occupational aspirations, were found to be strongly 
related to school success. 

Strength of influences on school learning 

Physical processes can often be explained as functions of a small number 
of variables interacting in simple ways. In contrast, schooling processes 
respond to a multitude of influences interacting in kaleidoscopic patterns. 
This research synthesis has confirmed that a large number of variables are 
moderately related to learning outcomes, but few if any single variables are 
very strongly related to learning. Authors of original research studies and of 
reviews and syntheses are appropriately cautious in stating the importance of 
particular items, and their caution is reflected in the relatively narrow range 
of mean ratings shown in Table 3. Nonetheless, taken together, the items 
examined in this synthesis are powerful determinants of school effects. 

The conclusions discussed in this section are based on what appears to be 
the most comprehensive analysis of the literature on effective educational 
practices for regular and special education. Related work involving the 
consensus of the panel of experts, the authors of research reviews, and regular 



and special practitioners adds further support to the conclusions (Wang, 
Walberg, & Reynolds, 1989; Reynolds, Wang, & Walberg, 1989). 

Still, certain caveats should be noted: It cannot be determined from the 

analyses, for example, what actual effect sizes will result; the analyses merely 
estimate their relative sizes. In addition, the analyses yield neither actual nor 
relative estimates of combinations of practices. It would seem reasonable to 
suppose that implementation of more practices with the highest estimates would 
yield the largest effects, but this supposition is a matter for subsequent 
empirical research. 

Another caveat applies to the content analysis of research literature on 
group-level effects, notably the literature on effective schools. Some of the 
effective schools factors have been analyzed in relation to school averages on 
achievement tests. Such relationships might be found somewhat larger or 
smaller if calculated for individual children. It can be expected that expert 
reviewers on this subject (on which the syntheses depend) would take this 
uncertainty into consideration in interpreting their findings. It has rarely 
been demonstrated that techniques that work for the average student have 
deleterious consequences for other students’ learning. 

Nonetheless, it is worth keeping this limitation in mind in interpreting 
the findings and in tracing their implications. There are many other cautions 
that ordinarily apply to educational research, such as the possibility that 
effective methods found a decade ago no longer apply today. These are 
obvious enough to leave to researchers and experienced educators as they 
think about how the findings apply in their own situations. 
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NOTES 



1. This research was supported in part by the Temple University Center for 
Research in Human Development and Education, and in part by a grant from 
the U. S. Department of Education’s Office of Special Education and 
Rehabilitative Services. The opinions expressed herein are solely those of the 
authors, and no official endorsement should be inferred. 

2. Copies of the detailed coding form and complete bibliographic citations for 
the 179 sources, as well as copies of the data archive are available from Dr. 
Margaret C. Wang at the Center for Research in Human Development and 
Education, Ritter Hall Annex, 9th Floor, Temple University, Philadelphia, PA 
19122. 

3. In addition to the coding and analysis of the 179 source documents, a 
survey was also conducted of the authors of all major source documents 
examined. The summary coding form described below was distributed to 
authors, with a request to provide overall ratings of the importance of the 228 
items to learning outcomes. A total of 78 forms were returned. These expert 
ratings were analyzed separately from the source document ratings, following 
identical procedures. Results were highly similar, with the exception that the 
experts generally tended to give somewhat higher numerical ratings. 

4. This panel included 12 prominent experts in areas of research on teaching, 
education, educational psychology, and special education. 

5. Most of these occurred for items in the scale, "History of Educational 
Placements," which accounts for the low mean of this variable in Table 3. 

6. If any supplementary items had been coded, these were reexamined as the 
forms were transcribed, and whenever possible were included under one of the 
prespecified items. This was generally possible because most supplementary 
items documented authors’ more detailed or specific empirical conclusions, for 
example, specific types of motivation related to learning, or particular variants 
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of instructional practices. Such detailed findings were incorporated into the 
broader variable prespecified on the form. The other major group of 
supplementary items were those documenting two-way or occasionally 
higher-order interactions. Because interactions represent more subtle findings 
and frequently fail to replicate, they were not transferred from the detailed 
form to the summary form. The summary forms were keyed and verified, and 
files were prepared for data analysis using standard statistical software 
packages. 

7. The weights used were equal to the numbers of original items included in 
the respective scales. Note that if there were no missing data, this procedure 
would result in giving all of the original items in a broad category equal 
weight. Where some items in a scale are missing, this procedure in effect 
assigns the mean of the nonmissing scale items to those missing observations. 
For any given scale, about 15 percent of the values of items on average were 
missing. 

8. As noted in the footnote to the table, these reported reliabilities are for 
means of all the items in a given category or scale. Due to missing data, 
values for some sources were based on means of fewer items. 

9. The highest ratings overall were assigned to "Psychomotor Variables," and a 
moderately high rating was also assigned to the scale "Accessibility Variables." 
However, only one item was included in each of these scales, and these items 
were referred to in six or fewer of the 179 sources analyzed. Thus, 
"Psychomotor Variables" and "Accessibility Variables" were set aside. The list 
of scales with the highest ratings include the 28 scales with more items and 
more ratings. 

10. A third item in this scale, "teacher provides explicit coaching to reduce 
aggression," also received a mean rating above 2.00, but was mentioned in only 
4 of the 179 sources. This item is of limited relevance in most regular 
educational settings. 
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Table 1 



Conceptual Framework with Illustrative Examples 



Category/Subcategory® 



Illustrative Variable 



Category L State and District Variables : These are variables associated with 
state and district level school governance and administration. They include 
state curriculum and textbook policies, testing and graduation requirements, 
and teacher licensure; as well as specific provisions in teacher contracts, and 
some district-level administrative and fiscal variables. 

District Level Demographics and School district size 

Marker Variables 

State Level Policy Variables Teacher licensure requirements 



Category IT. Out of School Contextual Variables : These are variables 
associated with the home and community contexts within which schools 
function. They include community demographics, peer culture, parental support 
and involvement, and amount of time students spend out-of-school on such 
activities as television viewing, leisure reading, and homework. 



Community Variables 



Peer Group Variables 



Home Environment and Parental 
Support Variables 

Student Use of Out of School Time 
Variables 



Socioeconomic level of 
community 

Level of peers’ academic 
aspirations 

Parental involvement in 
assuring completion of 
homework 

Student participation in clubs 
and extracurricular school 
activities 



Category III. School Level Variables : These are variables associated with 
school-level demographics, culture, climate, policies, and practices. They 
include demographics of the student body, whether the school is public or 
private, and levels of funding for specific categorical programs; school-level 
decision making variables, and specific school-level policies and practices, 
including policies on parental involvement in the school. 

Demographic and Marker Variables Size of school 




Teacher/Administrator Decision 
Making Variables 



Principal actively concerned 
with instructional program 



Table 1 (Category III, continued) 



Category/Subcategory ' 


Illustrative Variable 


School Culture Variables 

(Ethos Conducive to Teaching and 

Learning) 


School-wide emphasis on and 
recognition of academic 
achievement 


School-Wide Policy and Organizational 
Variables 


Explicit school-wide discipline 
policy 


Accessibility Variables 


Accessibility of educational 
program (overcoming 
architectural, communication, 
and environmental barriers) 


Parental Involvement Policy 
Variables 


Parental involvement in 
improvement and operation 
of instructional programs 



Category IV: Student Variables: These are variables associated with individual 
students themselves, including demographics, academic history, and a variety of 
social, behavioral, motivational, cognitive, and affective characteristics. 



Demographic and Marker Variables 


Gender 


History of Educational Placement 


Prior grade retentions 


Social and Behavioral Variables 


Positive, nondisruptive behavior 


Motivational and Affective Variables 


Attitude toward subject matter 
instructed 


Cognitive Variables 


Level of specific academic 
knowledge in subject area 
instructed 


Metacognitive Variables 


Comprehension monitoring 
(planning: monitoring effective- 
ness of attempted actions; 
monitoring outcomes of actions; 
testing, revising, and evaluating 
learning strategies) 


Psychomotor Variables 


Psychomotor skills specific to 
area instructed 



Table 1 (continued) 



Category/Subcategory 



Illustrative Variable 



Category V. Program Design Variables : These are variables associated with 
instruction as designed, and with the physical arrangements for its delivery. 
They include the instructional strategies specified by the curriculum, and 
characteristics of instructional materials. 

Demographic and Marker Variables Size of instructional group 

(whole class, small group, 
one-on-one instruction) 

Alignment among goals, 
contents, instruction, 
assignments, and evaluation 

Materials employ advance 
organizers 



Curriculum and Instructional 
Variables 

Curriculum Design Variables 



Category VI. Implementation. Classroom Instruction, and Climate Variables : 
These are variables associated with the implementation of the curriculum and 
the instructional program. They include classroom routines and practices, 
characteristics of instruction as delivered, classroom management, monitoring 
of student progress, and quality and quantity of instruction provided, as well 
as student-teacher interactions and classroom climate. 



Classroom Implementation Support 
Variables 



Classroom Instructional 
Variables 

Quantity of Instruction 
Variables 



Classroom Assessment 
Variables 



Classroom Management 
Variables 



Establishing efficient 
classroom routines and 
communicating rules and 
procedures 

Use of clear and organized 
direct instruction 

Time on task (amount of time 
students are actively engaged in 
learning) 

Use of assessment as a frequent, 
integral component of in- 
struction 

Group alerting (teacher uses 
questioning/ recitation strategies 
that maintain active partici- 
pation by all students) 



Table 1 (Category VI, continued) 



Category /Subcategory 



Illustrative Variable 



Student and Teacher Interactions: 
Social Variables 



Student and Teacher Interactions: 
Academic Variables 



Classroom Climate 
Variables 



Student responds positively to 
questions from other students 
and from teacher 

Frequent calls for extended, 
substantive oral and written 
response (not one-word answers) 

Cohesiveness (members of class 
are friends sharing common 
interests and values and 
emphasizing cooperative goals) 



‘Subcategories are listed below the description of each broad category, and are 
each illustrated with representative variables. For example, the first broad 
category includes two subcategories, "District Level Demographics and Marker 
Variables," and "State Level Policy Variables." 



Table 2 



Number and Percent of Source Documents by Type 



Type of Source® 


N 


Percent 


Total Pages 


Chapters from Annual Review Series 


86 


48 


3,179 


Handbook Chapters 


44 


25 


1,089 


Government Documents and 
Commissioned Reports 


20 


11 


772 


Book Chapters 


18 


10 


563 


Review Articles in Journals 


11 


6 


152 


Total 


179 


100 


5,755 



a A complete bibliography is available from the first author 
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Table 3 



Reliabilities, means, standard deviations, and frequencies for source ratings 



Category /Subcategory Reliability* 


Mean 


S.D. 


Frequency 


State and District Variables 


.90 


1.22 


.81 


27 


District demographics & marker vars. 


.95 


1.46 


.50 


14 


State level policy variables 


N.C. 


1.24 


1.00 


19 


Out of School Contextual Variables 


.99 


1.87 


.39 


59 


Community variables 


N.C. 


1.80 


.41 


15 


Peer group variables 


.98 


2.00 


.34 


18 


Home environment & parental support 


.95 


1.90 


.40 


47 


Student use of out-of-school time 


N.C. 


1.94 


.46 


17 


School Level Variably 


.95 


1.54 


.96 


102 


Demographics and marker variables 


.91 


1.74 


.56 


25 


Teacher/administrator decision making 


.87 


1.65 


.95 


21 


School culture variables 


.87 


1.84 


.43 


49 


School-wide policies and organization 


.76 


1.40 


1.14 


74 


Accessibility variables 


N.C. 


2.00 


.00 


2 


Parental involvement policy variables 


N.C. 


1.67 


.56 


23 


Student Variables 


.92 


1.83 


.57 


155 


Demographics and marker variables 


.71 


1.70 


.77 


90 


History of educational placements 


N.C. 


0.16 


1.80 


19 


Social and behavioral variables 


.80 


1.98 


.34 


35 


Motivational and affective variables 


.91 


1.93 


.42 


81 


Cognitive variables 


.88 


1.98 


.33 


101 


Metacognitivie variables 


.91 


2.08 


.36 


76 


Psychomotor variables 


N.C. 


2.33 


.52 


6 


Program Design Variables 


.90 


1.90 


.38 


142 


Demographic and marker variables 


N.C. 


1.97 


.54 


23 


Curriculum and instruction variables 


.90 


1.92 


.46 


108 


Curriculum design variables 


.89 


1.88 


.34 


97 


Classroom Instruction and Climate Variables 


.97 


1.84 


.66 


165 


Classroom implementation support 


.85 


1.84 


.38 


66 


Classroom instructional variables 


.89 


1.85 


.74 


156 


Quantity of instruction variables 


.94 


2.02 


.64 


69 


Classroom assessment variables 


N.C. 


1.89 


.30 


61 


Classroom management variables 


.98 


2.07 


.23 


42 


Student/teacher interactions: Social 


.73 


2.02 


.41 


44 


Student/teacher interactions: Academic 


.77 


1.89 


.44 


29 


Classroom climate variables 


.99 


2.01 


.38 


75 



*Coefficient alpha reliabilities were estimated for each scale from average variances and 
inter-item covariances. Due to missing data, ratings for some cases are based on fewer 
items. Thus, obtained reliabilities are somewhat lower than the figures reported in this 
table. "N.C." indicates values that were not calculable, either becaiise scales, consisted of 
only a single item, or due to patterns of missing data. 
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Variables Important to Learning: 
A Consensus From the Field 



This pamphlet provides a lists of all variables which emerged from a "meta-review" of pro- 
fessional literature concerned with variables that are important to school learning, as well as a 
summary of consensus ratings from the field. 

The variables are organized under six major categories, as follows: a)State and District Vari- 
ables, b) Out of School Contextual Variables, c) School Level Variables, d) Student Variables, e) Pro- 
gram Design Variables, and 0 Implementation, Classroom Instruction, and Climate Variables. These 
variables constitute the items included in a questionnaire which was sent to six professional groups 
(i.e., researchers, policy makers, special and regular education teachers, school psychologists, and 
principals). In total, 1,123 persons responded to the questionnare. The respondents were asked to 
rate each of the variable items according to a 3-point scale. 

Variables rated by the entire group as being of high importance ( rating of 2.6 and above), 
and of moderate importance (rating of 2.0 - 2.5) in arranging learning environments are indicated 
under the column "Number of Effective Practices. " 





Variables Important To Learning: 
A Consensus From the Field 



Number of Variables in 
Variables Each Variable Category 


Number of Effective Practices (rated as 
Important) In Each Variable Category 


CATEGORY I: State and District Variables 


A. District Level Demographics and Marker Variables 


(10) 


3 


B. State Level Policy Variables 


(6) 


3 


CATEGORY A: Out of School Contextual Variables 


A. Community Variables 


(3) 


3 


B. Peer Group Variables 


(5) 


5 


C. Home Environment and Parental Support Variables 


(9) 


9 


D. Student Use of Out of School Time Variables 


(5) 


3 


CATEGORY ID: School Level Variables 


A. Demographic and Marker Variables 


(8) 


3 


B. Teacher/ Administrator Decision Making Variables 


(6) 


6 


C. School Culture Variables (Ethos Conducive to 


(8) 


8 


Teaching and Learning) 


(13) 




D. School -Wide Policy and Organizational Variables 


(1) 


11 


E. Accessibility Variables 


(2) 


1 


F. Parental Involvement Policy Variables 




2 


CATEGORY IV: Student Variables 






A. Demographic and Marker Variables 


(7) 


4 


B. History of Educational Placements 


(3) 


3 


C. Social and Behavioral Variables 


(5) 


5 


D. Motivational and Affective Variables 


(9) 


9 


E. Cognitive Variables 


(12) 


12 


F. Metacognitive Variables 


(4) 


4 


G. Psychomotor Variables 


(1) 


1 


CATEGORY V: Program Design Variables 


A. Demographic and Marker Variables 


(4) 


4 


B. Curriculum and Instructional Variables 


(15) 


15 


C. Curriculum Design Variables 


(13) 


13 


CATEGORY VI: Implementation. Classroom Instruction 
and Climate Variables 


A. Classroom Implementation Support Variables 


(6) 


4 


B. Classroom Instructional Variables 


(26) 


26 


C. Quantity of Instruction Variables 


(12) 


11 


D. Classroom Assessment Variables 


(4) 


4 


E. Classroom Management Variables 


(5) 


5 


F. Student and Teacher Interactions: Social Variables 


(6) 


6 


G. Student and Teacher Interactions: Academic Variables 


(5) 


5 


H. Classroom Climate Variables 


(15) 


15 
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